FIG. 1 is a conceptual diagram illustrating a general transmission control protocol (TCP) state transition operation, showing an entire standard finite state machine used to better describe TCP for establishing and terminating a TCP connection. In FIG. 1, all states and changes are illustrated for accuracy.
First, TCP connection states are defined as follows:
LISTEN: a state in which a daemon of a server starts up and waits for a connection request;
SYN_SENT: a state in which a local client application has requested connection to a remote host;
SYN_RCVD: a state in which a server received a connection request from a remote client and replied to the client but has not yet received an acknowledgment message from the client;
ESTABLISHED: a state in which a server and a client are connected with each other after three-way handshaking is completed;
FIN_WAIT—1, CLOSE_WAIT, FIN_WAIT—2: states in which a server requests a client to terminate a connection, receives a reply, and thereby terminates the connection;
CLOSING: an uncommon state mainly in which an acknowledgment message is lost during transmission;
TIME_WAIT: a state in which a connection has been terminated but a socket temporarily remains open for slow segments which might have been lost; and
CLOSED: a state in which a connection is completely terminated.
According to the connection principle of TCP communication establishing and terminating a TCP connection, a client and a server must go through a three-way handshaking procedure to establish a TCP connection, as defined in previous documents, e.g., Transmission Control Protocol, RFC 793, Jon Postel, DARPA Internet Program Protocol Specification, September 1981.
In other words, for TCP communication between a client and a server, the process of exchanging a connection request (synchronous (SYN)) packet, a connection-request acknowledgment (synchronous/acknowledge (SYN/ACK)) packet, and an acknowledgment (ACK) packet is absolutely necessary.
Here, a successful three-way handshaking procedure of connection request (SYN), connection-request acknowledgment (SYN/ACK), and acknowledgment (ACK) indicates that a TCP connection between a client and a server is established, that is, a TCP communication port of the server requested by the client is open for communication.
However, increase in the number of companies providing services via the Internet has been accompanied by increased attacks abusing the three-way handshaking procedure, e.g., denial of service (DoS) attacks such as SYN flooding attacks.
This will be described with examples. First, when a server receives a finish (FIN) packet in which a FIN flag is set from a malicious client immediately after receiving a SYN packet in which a SYN flag is set, the TCP state of the server directly jumps from the SYN_RCVD state to the CLOSE_WAIT state. For this reason, the server sends an ACK packet, in which an ACK flag corresponding to the FIN packet is set, to the client and waits for an ACK packet of the client in reply to the sent ACK packet.
Second, when a server receives a SYN packet of a client, sends its SYN/ACK packet, in which SYN and ACK flags are set, to the client in order to accept the connection request, and then cannot receive an ACK packet corresponding to the SYN/ACK packet from the client, the server waits in the SYN_RCVD state until the ACK packet is received. In this way, when the malicious client does not complete the procedure after continuously sending the packet to the server, the queue of the server overflows, and thus a DoS event happens.
Third, by a SYN flooding attack, i.e., an attack of continuously sending SYN packets alone from a client, a server continuously waits in the SNC_RCVD state. For this reason, the SYN packets are continuously stacked in the queue of the server. In result, all the queues fill up and the server cannot receive any connection request.
Conventional TCP state transition supports all operations of an operating system (OS) in some cases, and in other cases does not, depending on the configuration of the OS. In particular, a DoS attack such as a SYN flooding attack places a heavy load on a server on service because the server does not perform the checking step of switching to all states suggested in an initial three-way handshaking procedure. Consequently, the server becomes unable to provide service and frequently crashes.